A Study of Supervised Spam Detection applied to Eight Months of Personal E-Mail

In the last year or two, unwelcome email has grown to the extent that it is inconvenient, annoying and wasteful of computer resources. More significantly, its volume threatens to overwhelm our ability to recognize welcome messages, and hence to destroy our trust in email as a reliable communication medium. An automatic spam filter can mitigate these problems, provided that it acts in a reliable and predictable manner. We evaluate ten spam detection methods embodied in six popular open-source spam filters by applying each method sequentially to all of the e-mail received by one individual (X) from August 2003 through March 2004. These 49,086 messages were originally judged in real-time by X. The messages and judgements were recorded, and reproduced so as to provide the same evaluation suite for all the methods. Five of the methods are derived from Spamassassin [spamassassin.org 2004], a hybrid system which includes both static spam-detection rules and a Bayesian statistical learning component. The purpose of intra-Spamassassin comparison is evaluate the relative contributions of the static and learning components in various configurations. The other methods are all “pure” statistical learning systems, in that they contain essentially no spam-detection rules. We compare these – Bogofilter [Raymond 2004], CRM-114 [Yerazunis 2004a], DSPAM [Zdziarski 2004], SpamBayes [Peters 2004], and Spamprobe [Burton 2002a] – against each other and against the learning component of Spamassassin. Recent evaluations have reported very high accuracies – some higher than 99.9% – for several of these filters [Burton 2002a; Zdziarski 2004; Yerazunis 2004a; Louis 2004; Yerazunis 2004b; Holden 2004]. We contrast these studies and others [Sahami et al. 1998; Graham 2004; Robinson 2004; Androutsopoulos et al. 2000; Tuttle et al. 2004] with ours following the presentation of our results. Our study’s objective is to provide a controlled, realistic, statistically meaningful evaluation of several common filters and the methods they embody. While our study is limited to the extent that X’s email is typical, and to the extent that the ten subject methods represent the state of the art, we present our methods and analysis in sufficient detail that they may be reproduced with other email streams and filter implementations. In addition, we have archived our evaluation suite so that we may use it to evaluate future spam detection methods. Spam filtering may be effected in a number of configurations, which we categorize as: manual, static, unsupervised, and supervised. With manual filtering (see figure 1), the task falls to the email recipient who must examine and classify each message that is received. As mentioned above, this process is tedious and error-prone,